Beyond the word and image: characteristics of a common meaning system for language and vision revealed by functional and structural imaging

This research tests the hypothesis that comprehension of human events will engage an extended semantic representation system, independent of the input modality (sentence vs. picture). To investigate this, we examined brain activation and connectivity in 19 subjects who read sentences and viewed pictures depicting everyday events, in a combined fMRI and DTI study. Conjunction of activity in understanding sentences and pictures revealed a common fronto-temporo-parietal network that included the middle and inferior frontal gyri, the parahippocampal-retrosplenial complex, the anterior and middle temporal gyri, the inferior parietal lobe in particular the temporo-parietal cortex. DTI tractography seeded from this temporo-parietal cortex hub revealed a multi-component network reaching into the temporal pole, the ventral frontal pole and premotor cortex. A significant correlation was found between the relative pathway density issued from the temporo-parietal cortex and the imageability of sentences for individual subjects, suggesting a potential functional link between comprehension and the temporo-parietal connectivity strength. These data help to define a "meaning" network that includes components of recently characterized systems for semantic memory, embodied simulation, and visuo-spatial scene representation. The network substantially overlaps with the "default mode" network implicated as part of a core network of semantic representation, along with brain systems related to the formation of mental models, and reasoning. These data are consistent with a model of real-world situational understanding that is highly embodied. Crucially, the neural basis of this embodied understanding is not limited to sensorimotor systems, but extends to the highest levels of cognition, including autobiographical memory, scene analysis, mental model formation, reasoning and theory of mind.